Multi data path config can cause a shard to be perceived as corrupted #4674
Comments
Multi data path config support writes a file to a data location based on the available size (by default). There is a Lucene file called segments.gen that has the same name, and only in that case, we need to make sure we alway write it to the same data location, otherwise, the index will have multiple segments.gen files, and the shard can seem to be corrupted. The message if this case happens is that segments_xxx file was not found, in which case, a find for segments.gen can yield multiple files. Deleting the segments.gen files will cause the shard to recover properly (as its an extra protection layer to resolve the segments header by Lucene) Make sure the segments.gen file is writtne to the same directory every time fixes elastic#4674
Multi data path config support writes a file to a data location based on the available size (by default). There is a Lucene file called segments.gen that has the same name, and only in that case, we need to make sure we alway write it to the same data location, otherwise, the index will have multiple segments.gen files, and the shard can seem to be corrupted. The message if this case happens is that segments_xxx file was not found, in which case, a find for segments.gen can yield multiple files. Deleting the segments.gen files will cause the shard to recover properly (as its an extra protection layer to resolve the segments header by Lucene) Make sure the segments.gen file is writtne to the same directory every time fixes #4674
|
Hi, I was using ES 0.90.7 and my shards were after restart of the cluster going missing. Physically, I can see the files (in 6 cases out of 8) still on the disk, but the shards won't come up with the indication of a segments_X file missing. I have followed your advice of removing the segments.gen file in order to recover, but the shards are not coming up. Even after restart of the cluster, the shards are not coming up, the error is now: [2014-01-15 18:46:00,504][DEBUG][cluster.service ] [Synch] processing [shard-failed ([1millionnewv2][4], node[fWigetX5QNar2zIpvkEK_Q], [P], s[INITIALIZING]), reas What can I do if the usual strategy of the segments.gen file removal doesn't work? I have even updated the ES to the latest 0.90.10 version but the restart problems are still the same and the shards are not coming up. I have lost a considerable amount of data, in those 2 shards that were wiped out, but I could still save a lot of data in the 6 shards I am could recover. |
Multi data path config support writes a file to a data location based on the available size (by default). There is a Lucene file called segments.gen that has the same name, and only in that case, we need to make sure we alway write it to the same data location, otherwise, the index will have multiple segments.gen files, and the shard can seem to be corrupted. The message if this case happens is that segments_xxx file was not found, in which case, a find for segments.gen can yield multiple files. Deleting the segments.gen files will cause the shard to recover properly (as its an extra protection layer to resolve the segments header by Lucene) Make sure the segments.gen file is writtne to the same directory every time fixes elastic#4674
|
@sbarton are you sure you deleted the segments.gen from all data directories for the relevant shard ([1millionnewv2][4]) (or, better yet, just delete it recursively across all data locations)? the failure suggests you potentially didn't. |
|
@kimchy I have made sure I removed the segments.gen from all data directories (using find command) but still no luck. I at the end gave up (I had more shards in the same situation on other machines - tried the same approach but none of them came back) and re-indexed the whole thing once again. But I can say, that the 0.90.10 version is not loosing shards even after several harsh restarts. |
Multi data path config support writes a file to a data location based on the available size (by default). There is a Lucene file called segments.gen that has the same name, and only in that case, we need to make sure we alway write it to the same data location, otherwise, the index will have multiple segments.gen files, and the shard can seem to be corrupted. The message if this case happens is that segments_xxx file was not found, in which case, a find for segments.gen can yield multiple files. Deleting the segments.gen files will cause the shard to recover properly (as its an extra protection layer to resolve the segments header by Lucene) Make sure the segments.gen file is writtne to the same directory every time fixes elastic#4674
Multi data path config support writes a file to a data location based on the available size (by default). There is a Lucene file called
segments.genthat has the same name, and only in that case, we need to make sure we alway write it to the same data location, otherwise, the index will have multiple segments.gen files, and the shard can seem to be corrupted.The message if this case happens is that segments_xxx file was not found, in which case, a find for segments.gen can yield multiple files. Deleting the segments.gen files will cause the shard to recover properly (as its an extra protection layer to resolve the segments header by Lucene)
The text was updated successfully, but these errors were encountered: